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Scientific  Progress 


Our  scientific  progress  has  centered  around  three  projects  that  focus  on  different  aspects  of  our  proposed  work.  We  describe 
each  below,  focusing  on  our  achievements  in  the  final  reporting  period. 

Building  Hierarchies  of  Concepts  via  Crowdsourcing 

Hierarchies  of  concepts  and  objects  are  useful  across  many  real-world  applications  and  scientific  domains.  Task-dependent 
hierarchies  are  expensive  and  time-consuming  to  construct.  They  are  usually  built  in  a  centralized  manner  by  a  group  of  domain 
experts.  This  process  makes  it  infeasible  to  create  separate  hierarchies  for  each  specific  domain.  Attempts  to  build  hierarchies 
using  fully  automatic  methods  have  failed  to  capture  the  relationships  between  concepts  as  perceived  by  people.  The  resulting 
hierarchies  perform  poorly  when  deployed  in  real-world  systems.  With  the  recent  popularity  of  crowdsourcing  platforms,  such  as 
Amazon  Mechanical  Turk  (AMT),  non-expert  workers  (the  crowd)  can  be  employed  at  scale  and  low  cost,  to  build  hierarchies 
guided  by  their  knowledge.  Most  existing  methods  as  well  as  methods  employing  domain  experts  usually  generate  only  a  single 
hierarchy  aiming  to  best  explain  the  data  or  the  semantic  relationships  among  the  concepts.  This  ignores  the  natural  ambiguity 
and  uncertainty  that  may  exist  in  the  semantic  relationships. 

In  this  work,  we  introduced  a  novel  approach  to  learning  hierarchies  over  concepts  using  crowdsourcing.  Our  approach 
incorporates  simple  questions  that  can  be  answered  by  non-experts  without  global  knowledge  of  the  concept  domain.  To  deal 
with  the  inherent  noise  in  crowdsourced  information,  people’s  uncertainty,  and  possible  disagreement  about  hierarchical 
relationships,  we  develop  a  Bayesian  framework  for  estimating  posterior  distributions  over  hierarchies.  When  new  answers 
become  available,  these  distributions  are  updated  efficiently  using  a  sampling  based  approximation  for  the  intractably  large  set 
of  possible  hierarchies.  The  Bayesian  treatment  also  allows  us  to  actively  generate  queries  that  are  the  most  informative  given 
the  current  uncertainty.  New  concepts  can  be  added  to  the  hierarchy  at  time  point,  automatically  triggering  queries  that  enable 
the  correct  placement  of  these  concepts. 

We  investigated  several  aspects  of  our  framework  and  demonstrated  that  it  is  able  to  recover  high-quality  hierarchies  for  real 
world  concepts  using  AMT.  Importantly,  by  reasoning  about  uncertainty  over  hierarchies,  our  approach  is  able  to  unveil 
confusion  of  non-experts  over  concepts,  such  as  whether  tomato  is  a  fruit  or  vegetable,  or  whether  the  wrist  is  part  of  a  person’s 
arm  or  hand.  We  believe  that  these  abilities  are  extremely  useful  for  applications  where  hierarchies  should  reflect  the 
knowledge  or  expectations  of  regular  users,  rather  than  domain  experts.  This  work  was  published  in  IJCAI  2015  and  the  paper 
is  uploaded  as  part  of  this  final  report. 

Robot  Programming  by  Demonstration  with  Situated  Spatial  Language  Understanding 

General-purpose  robots  that  can  manipulate  everyday  object  could  take  on  many  useful  tasks  in  human  environments. 

However,  programming  such  robots  to  robustly  function  in  every  possible  environment,  with  every  possible  user  is  extremely 
challenging.  Instead,  our  research  seeks  to  develop  robots  that  can  be  programmed  by  their  end-users  to  function  in  their 
particular  environment. 

Programming  by  Demonstration  (PbD)  is  an  intuitive  method  that  enables  users  to  program  new  capabilities  on  a  robot  simply 
by  demonstrating  the  desired  behavior.  Kinesthetic  teaching  (i.e.,  physically  moving  the  robot  through  desired  states)  allows 
users  to  directly  demonstrate  desired  arm  configurations  and  motions  at  a  fine  grained  level  and  is  often  the  most  efficient 
interface  for  PbD.  However,  it  requires  physical  effort  from  the  user  and  it  is  not  accessible  to  persons  with  certain  motor 
impairments.  Furthermore,  it  requires  the  user  to  be  co-located  with  the  robot.  In  this  paper,  we  present  a  natural  language 
interface  for  programming  a  robot  to  perform  manipulation  tasks  purely  through  verbal  commands.  We  exploited  the  fact  that 
task-oriented  motion  is  often  relative  to  landmarks  in  the  environment  and  that  it  can  be  described  with  spatial  language  (e.g., 
object  reference  expressions  and  prepositions). 

We  implemented  a  set  of  robust  parametrized  motion  procedures  on  a  PR2  mobile  manipulator  and  we  presented  a  situated 
language  understanding  model  that  takes  a  natural  language  utterance  and  infers  the  intended  procedure  and  its  parameter 
instantiation.  We  demonstrated  that  our  method  allows  programming  different  manipulation  tasks,  even  with  ambiguous 
language  and  is  on  par  with  kinesthetic  programming  in  terms  of  efficiency.  This  work  was  published  in  ICRA  2015  and  the 
paper  is  uploaded  as  part  of  this  final  report. 

Scalable  Semantic  Parsing  with  Partial  Ontologies 

Recently,  significant  progress  has  been  made  in  learning  semantic  parsers  for  large  knowledge  bases  (KBs)  such  as  Freebase 
(FB)  (Cai  and  Yates,  2013;  Berant  et  al.,  2013;  Kwiatkowski  et  al.,  2013;  Reddy  et  al.,  2014).  Although  these  methods  can  build 
general  purpose  meaning  representations,  they  are  typically  evaluated  on  question  answering  tasks  and  are  designed  to  only 
parse  questions  that  have  complete  ontological  coverage,  in  the  sense  that  there  exists  a  logical  form  that  can  be  executed 
against  Freebase  to  get  the  correct  answer.  In  this  project,  we  instead  consider  the  problem  of  learning  semantic  parsers  for 
open  domain  text  containing 

concepts  that  may  or  may  not  be  representable  using  the  Freebase  ontology.  Sich  an  approach  has  wide  application,  including 


especially  building  robust  parsers  for  understanding  robot  directed  utterance  in  human-robot  interaction  scenarios. 


Even  very  large  knowledge  bases  have  two  types  of  incompleteness  that  provide  challenges  for  semantic  parsing  algorithms. 
They  (1 )  have  partial  ontologies  that  cannot  represent  the  meaning  of  many  English  phrases  and  (2)  are  typically  missing  many 
facts.  They  include  subjective  or  otherwise  unmodeled  phrases  such  as  “relaxed”  and  “quake-hit.”  Freebase,  despite  being 
large-scale,  contains  a  limited  set  of  concepts  that  cannot  represent  the  meaning  of  these  phrases.  They  also  refer  to  entities 
that  may  be  missing  key  facts.  For  example,  a  recent  study  (West  et  al.,  2014)  showed  that  over  70%  of  people  in  FB  have  no 
birth  place,  and  99%  have  no  ethnicity.  In  our  work,  we  introduce  a  new  semantic  parsing  approach  that  explicitly  models 
ontological  incompleteness  and  is  robust  to  missing  facts,  with  the  goal  of  recovering  as  much  of  a  sentence’s  meaning  as  the 
ontology  supports.  We  argue  that  this  will  enable  the  application  of  semantic  parsers  to  a  range  of  new  tasks,  such  as 
information  extraction  (IE)  and  robotic  dialog  systems,  where  phrases  rarely  have  full  ontological  support  and  new  facts  must  be 
added  to  the  KB. 

To  better  model  such  text,  we  presented  a  new  semantic  parser  that  builds  logical  forms  with  concepts  from  a  target  ontology 
and  open  concepts  that  are  introduced  when  there  is  no  appropriate  concept  match  in  the  target  ontology.  Figure  2  shows 
examples  of  the  meanings  that  we  extract.  Only  the  first  of  these  examples  can  be  fully  represented  using  Freebase,  all  other 
examples  require  explicit  modeling  of  open  concepts.  To  build  these  logical  forms,  we  follow  recent  work  for  Combinatory 
Categorical  Grammar  (CCG)  semantic  parsing  with  Freebase  (Kwiatkowski  et  al.,  2013),  extended  to  model  when  open 
concepts  should  be  used.  We  develop  a  two-stage  learning  algorithm:  we  first  compute  broad  coverage  lexical  statistics  over  all 
of  the  data,  which  are  then  incorporated  as  features  in  a  full  parsing  model.  The  parsing  model  is  tuned  on  a  hand-labeled  data 
set  with  gold  analyses. 

Experiments  demonstrated  the  benefits  of  the  new  approach.  It  significantly  outperforms  strong  baselines  on  both  a  referring 
expression  resolution  task,  where  much  like  in  the  OA  setting  we  directly  evaluate  if  we  recover  the  correct  logical  form  for  each 
input  noun  phrase,  and  on  entity  attribute  extraction,  where  individual  facts  are  extracted  from  the  groundable  part  of  the  logical 
form.  We  also  see  that  modeling  incompleteness  significantly  boosts  precision;  we  are  able  to  more  effectively  determine  which 
words  should  not  be  mapped  to  KB  concepts.  When  run  on  all  of  the  Wikipedia  category  data,  we  estimate  that  the  learned 
model  would  discover  12  million  new  facts  that  could  be  added  to  Freebase  with  72%  precision. 

Technology  Transfer 


